A database is a structure for storing and relating data within e.g., a computer system. Different database architectures exist depending on the intended usage. The primary use for general purpose databases is to manage and facilitate data entry and retrieval in relation to the relevant application. A recent trend has been the emergence of specialized database architectures optimized to work with specific application domains.
Complex event processing (CEP) is a technology for low-latency filtering, correlating, aggregating, and/or computing on real-world event data. Such data is usually generated at high frequencies and so needs to be saved in an appropriate database to allow it to be evaluated, whether in real time, or at a later stage. Several specialized database products have emerged which attempt to store such data, which is generated in quantities that normally overwhelm general purpose databases.
The following products are available for use in CEP applications, and provide different functionalities for manipulating CEP data.
ProductDescriptionTechnologyVhayu velocity High performance proprietary Proprietary, non-database optimized to work relational in-memorywith high-frequency financial databasemarket dataKX systemsHigh performance database Optimized, column-basedKDB+to monitor real-time events databaseand detect and report faults for data-intensive applicationsStreamBaseEvent processing platform Integrated developmentwhich allows for development environment along withof applications that query and specialized compileranalyze high-volume real-time data streams
These products aim to provide improvement of both underlying database technologies and processing capabilities. However, data storage and querying or retrieval of the data is still carried out according to conventional processes. While these databases are well-suited to performing traditional transaction-oriented operations, they do not provide an efficient means for allowing large amounts of contiguous data to be accessed and/or evaluated, other than standard querying methods.
Such requests for large amounts of contiguous data are relevant to the provision of descriptive statistics where the importance of individual records is less than the overall description. Descriptive statistics are now becoming increasingly important especially for high-frequency high-volume data applications.
The process of evaluating large contiguous datasets is central to responding to statistical descriptive data requests.
The financial services community consists of data providers and clients. Data providers deal with both large institutional clients (e.g., banks) and smaller clients (e.g., retail investors). Dealing with the larger clients is either done directly or through 3rd party vendors (e.g., Vhayu) to provide them with all market data in order to allow for the construction of sophisticated and accurate statistical variables. However, at present this is not possible with smaller clients due to costs associated with the large bandwidth and computational requirement needed for delivering the complete market feed. Therefore, smaller clients are only provided with market snapshots or summaries, which only allow for variable approximations.